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Abstract 


The extended trust region subproblem has been the focus of several 
research recently. Under various assumptions, strong duality and certain 
SOCP/SDP relaxations have been proposed for several classes of it. Due to 
its importance, in this paper, without any assumption on the problem, we 
apply the widely used alternating direction method of multipliers (ADMM) 
to solve it. The convergence of ADMM iterations to the first order stationary 
conditions is established. On several classes of test problems, the quality of 
the solution obtained by the ADMM for medium scale problems is compared 
with the SOCP/SDP relaxation. Moreover, the applicability of the method 
for solving large scale problems is shown by solving several large instances. 


Keywords: Extended trust region subporblem; Alternating method; Non- 
convex optimization; Semidefinite program; Second order cone program. 


1 Introduction 


Consider the following extended trust region subproblem with m linear in- 
equalities (m-eTRS): 
1 
min xt Ax tata 
liell? <1, (m-eTRS) 
pesky t=1 
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where A € R”*” is a symmetric matrix but not necessarily positive definite, 
a,b; € R” and 6; € R. Due to its importance and crucial role in solving 
general nonlinear optimization problems, several versions of it have been the 
focus of current research [6,7,11,13,17,18]. In [13], the authors have shown 
that the dimension condition, dim( Ker(A — AiJ,)) > s+ 1, where , is 
the smallest eigenvalue of A and s = dim(span{by, .., 0m }), together with the 
Slater condition ensure that a set of combined first and second-order Lagrange 
multiplier conditions are necessary and sufficient for the global optimality of 
(m-eTRS) and consequently for strong duality. In [11], the authors have pro- 
posed an induction technique that reduces (m-eTRS) to several small-sized 
trust region subproblems (TRS). When m is not too large, their method 
can be very efficient but requires finding local-nonglobal minimum (LNGM), 
which no efficient algorithm is known to find it. Moreover, no numerical ev- 
idences are reported by the authors to support their theoretical foundation. 
Also they have improved the dimension condition by Jeyakumar and Li un- 
der which (m-eTRS) admits an exact SDP relaxation. They proposed the 
following condition 


rank ({A =; Arn by bo sie bm) < n—1. (1) 


This rank condition implies that the global optimal solution of the (m-eTRS) 
does not happen at the (LNGM) of (TRS) [8,15]. In a most recent study [7], 
the authors have proposed the following SOCP/SDP relaxation for (m-eTRS) 
based on the following assumption: 

Assumption 1: For all i < j, there exists no feasible point x for (m-eTRS) 
such that bf 2 = 6; and bf x = 5. 


1 
Ming, x girace(AX) t+alz 
s.t. trace(X) <1, X > aa’, 
BiB; — Bjo, z — Byby a+b; Xb; >0, 1<i<j<m. 


They also have proved that this relaxation is tight when we consider the fol- 
lowing assumption instead of Assumption 1. 

Assumption 2: For all 7 < j, there exists no x with ||z|| < 1 such that 
bla = 6; and by x = 8). 


Moreover, in several recent studies, when m = 1, efficient algorithms have 
been proposed to solve large instances of (m-eTRS) [17,18]. Therefore, find- 
ing an efficient algorithm to solve (m-eTRS) problems when m > 1 is still 
an active research area as the proposed methods either are using certain as- 
sumptions to simplify the analysis or using the SOCP/SDP relaxation, which 
both are not applicable practically, specially for large scale instances. 
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Recently, the Alternating Direction Method of Multipliers (ADMM) have 
been widely used to solve optimization problems arising in machine learning, 
signal processing, matrix factorization, financial optimization and etc. [2,3,5, 
12,14,19,22]. Although the method exhibits faster convergence in practice, its 
global convergence is still the subject of research. Under various assumptions 
on the sequence generated by the method like ”if the limit point exists” 
or "if the Lagrange multipliers are bounded”, its global convergence to a 
stationary point is established in the above-mentioned papers. In this paper, 
we apply ADMM to (m-eTRS) without any assumption on the geometry 
of the feasible region. The convergence of the method to the first order 
necessary optimality conditions is proved. Finally, on several medium scale 
instances its performance is compared with the SOCP/SDP relaxation of [7] 
and then several large instances also are solved by ADMM. To the best of our 
knowledge, there is no algorithm in the literature for large scale (m-eTRS) 
which we could provide comparison. 


2 ADMM for (m-eTRS) 


One can write (m-eTRS) in the following equivalent form: 


1 

min 5 Ae +alaz 
IIe|/? <1, (2) 
biz < Bi, a=1,-:- ,™m 


L= Zz. 


Now consider the following augmented Lagrangian for (2): 


1 
L(a,z,r) = st Aw tala+ dT (x —2z)4 ale rales 


where ),;’s are Lagrange multipliers and p > 0 is the penalty parameter. The 
ADMM iterations for the given x* and * are as follows [5]: 


e Step 1: 2*+! = argmingr,<g,, i=1,-- L(a*,z,d*). 


75m 


e Step 2: 2*t! = argmin).\)2<)L(a, 2**1, ). 


e Step 3: A*t! = \*¥ + yp(x**+1 — z*+1), where 7 € (0,1) is a constant. 


In what follows, we discuss the above steps. In Step 1, we need to solve the 
following convex quadratic optimization problem with m linear inequality 
constraints which can be efficiently solved using existing convex optimization 
software packages like CVX [9]: 
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min $272 —(A* + pr*)?z 
bP z < B;, i=1,---,m. (3) 


In Step 2 we need to solve the following (TRS) problem: 


min 527 (A + pl)z+ (a+ r* — p2*t}) Pa 
llz|[? <1. (4) 


The (TRS) problems are widely used and studied in the literature [8] and 
they can be solved efficiently using the exiting eigenvalue approaches even 
for large instances [1,16]. Now, the ADMM algorithm for solving (m-eTRS) 
can be outlined as follows. 


ADMM Algorithm for solving (m-eTRS) 


Input parameters tol > 0, maxiter> 0. Choose appropriate penalty pa- 
rameter p > 0 and y > 0. Set k = 0 and choose appropriate x* and \* 

For k = 1,--- , maxiter do 

Solve quadratic optimization problem (3) and let its solution be z**?. 
Solve the (TRS) problem (4) and let its solution be «*t?. 

If ||2*+1 — 2*+1|| < tol, then exit with x**1 as output. 

end if 
Set AP+1 = \F 4 yo(ePt! — zk+1) and k=k+1. 
end for. 


As we see, the method is very easy to implement and subproblems of the 
Steps 1 and 2 are efficiently solvable even for large instances. In what follows, 
we discuss the convergence of the above algorithm to the stationary point of 
(m-eTRS). First we present the following lemma. 


Lemma 1. Suppose that {A*} is bounded and TP, \|A**! — A*||?_ < oo. 
Then 
\jx**? — x*|| +0, |[2**+— 2*|| 50 as ko. 


Proof. Since x**! solves problem (4) at k-th iteration and a* — x*+1 is a 
feasible direction with respect to the feasible region of (4), then 
Vella? gL YT =o") > 0. (5) 


Now 
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(at _ g®t1)\T (4 + pl)(a* _ ght ty 


» 
+V,L(a*"1, anaes NP cae? a ea 2 Se ie = ese, (6) 


La ON) _ Tilak ght) = 


where the inequality follows from the definition of the smallest eigenvalue of 
A, Ai, and (5). We also have 


L(a*, 2°, *) — L(2*, 2**7, 4*) > 0, (7) 
as z*+! is the minimizer of L(x*,z,\*). On the other hand 
Leek! kt A) = Lak gktl be | = OF = METYE (gh tt _ ght) 
1 
Say P|? 8 
aol! | (8) 
Now using (5), (7), and (8) we have 


Lao OP) SE a ha a 2 
Le eh) — Diag yg *) a La. ad = Lal ger) 


nN 1 
S SP lah a* ||? aere Palla (9) 
YP 


Since {A*} and {x*} are bounded, then from Step 3 of ADMM iterations, 
{z*} also is bounded. Therefore {L(a*, z*, \*)} is bounded. Moreover, since 
by assumption 77°, ||A*t — A¥ ||? < oo, thus from (9), 9°, ||e*tt — *|/? 
is a bounded series (in the sense that the sequence of partial sums is 
bounded)with nonnegative terms, thus it is convergent. Then ||x* — 2**1|| = 
0, as k— oo. Moreover since by assumption ||\* — \**1|| > 0, as k > ov, 
from the Step 3 we have x* — z* +0, as k > oo. Finally since 


and we know z* — «* > 0, 2* —a2*t1 4 0, ehtl— zt _. 0, ask > &w, 
then z* — z*+1 _, 0. 


It should be noted the boundedness assumption of multipliers in the 
lemma is a standard assumption for convergence analysis of nonconvex op- 
timization problems [14]. Similar assumptions are used to prove the con- 
vergence to the stationary point in [21,22]. In what follows we prove the 
convergence to the first order stationary conditions 


Theorem 1. Let (x*,z*,X*) be any accumulation point of {(a*,z*, \*)} 
generated by the ADMM Algorithm. Then by boundedness of {*} and 
re, AST! = AP ||? < 00, x* satisfies the first order stationary conditions. 


Proof. Since (a*, z*, A*) is an accumulation point of {(2*, z*, \*)}, then there 
exists a subsequence {(ar*,z",\")},e7 that converges to (x*,2*,A*). Now 
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consider subproblems that should be solved in Steps 1 and 2. As we men- 


tioned, subproblem (3) in Step 1 is a convex quadratic optimization problem 
which its necessary and sufficient optimality conditions are as follows: 


m 
p2t! — (AP + pa*) + oS vk+1b, = 0, 


i=1 
DMG eSB) 0, be x Oy aS iat yas (10) 
where per ig are the Lagrange multipliers. Moreover, subproblem in Step 2 


is a (TRS) as given in (4) that we have the following necessary and sufficient 
optimality conditions for it: 


(A+ pln + OT las ON sag =—(a+ cS pe); 
pron ehh) 1) =o [atin Peal, (11) 
ANDi a Oho ES Od. 


Now by taking the limit of both (10) and (11), we get 


(A+ pln + 2p"In)2* = —(a+ r* — pz"), 
w*([[x*||? — 1) =0, |la*|[? <1, 


Pee et Tad ier |e (12) 
pz* — (* + px*) + So uf; =0, 
i=1 


vi (bP 2" — B,) =0, bf 2* < 6; 1=1,>++ ,m. 
From the first and forth equations of (12), we get 


(A+ 2p" Tn )ar* = —a —Y v7bi, 
i=1 


which with the second and the last equations are the first order stationary 
conditions. 


3 Numerical experiments 


In this section, we present several randomly generated test problems to as- 
sess the performance of ADMM for solving (m-eTRS). For small dimension 
problems, we compare ADMM with the SOCP/SDP relaxation (R,,). All 
computations are performed in MATLAB R2015a on a 2.50 GHz laptop with 
4 GB of RAM. To solve the SOCP/SDP reformulation, we have used CVX 
1.2.1. For all test problems, we set tol = 10~® and maxiter= 100. Our exten- 
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Table 1: Comparison between ADMM and SOCP/SDP relaxation for the first class of 
test problems with density=0.1. 


Dimension | Algorithm Time (s) 
n=100 ADMM 1.4 
SOCP/SDP 1.9 
n=300 ADMM 3.2 
SOCP/SDP | 274 
n=500 ADMM 4.8 
SOCP/SDP 163.5 


sive testing showed that y = 0.9 and p = —2, + 1 are appropriate choices. 
Moreover, we set \° = 2e, where e denotes the all one vector and x° = Woe 
Finally, to solve the (TRS) problems within the algorithm we have used the 
algorithm in [1] and to solve the quadratic optimization problems we have 


used the ’quadprog’ command of MATLAB. 


e First class of test problems: 

For this class we consider m = 2 in (m-eTRS), A = sprandsym(n, density), 
a =randn(n, 1), by = e1, be = —e1, the unit vector in R” and 6; = 0.1 
and (2 = 0.1. As we see, the two linear inequality constraints are par- 
allel, then the relaxation in [7] is exact. Results are summarized in 
Tables 1 to 3 for the average of 10 runs. In Tables 1 and 2 we compare 
ADMM with the SOCP/SDP relaxation for different densities. As we 
see, ADMM is much faster than SOCP/SDP relaxation. Moreover, for 
these two tables beside the time, we also have computed the relative 
objective function difference which is always below O(10~8) for all test 
problems. This numerically verifies that ADMM converges to the same 
solution as SOCP/SDP relaxation does. In Table 3 we just report the 
results of ADMM as the relaxation is not applicable for these prob- 
lems. In this table, KKT1 denotes the first order stationary condition, 
namely ||(A + 2u*I,)a* +a+ >>", ve bill. 


e Second class of test problems: 

For this class we consider m = 5, the number of linear inequalities. As 
before, we consider A = sprandsym(n, density) and a = randn(n, 1) 
and consider b = rand(n,m), x = randn(n,1) and 6 = . Obvi- 
ously, the feasible region of (m-eTRS) is nonempty. Results of apply- 
ing ADMM and SOCP/SDP relaxation to this class are summarized in 
Tables 4 to 6. Similar observations to the previous class hold here as 
well. 
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Table 2: Comparison between ADMM and SOCP/SDP relaxation for the first class of 
test problems with density=0.01. 


Dimension | Algorithm Time (s) 
n=100 ADMM 0.75 
SOCP/SDP | _L1 
n=300 ADMM 2.6 
SOCP/SDP 26.7 
n=500 ADMM 4.1 
SOCP/SDP | 141.5 


Table 3: Results of ADMM for the first class of test problems with density=0.001. 


Dimension | KKT1 Time (s) 
n=3000 8.78 x 10-4 21.5 
n=5000 5.51 x 10-4 37.7 
n=8000 1.03 x 10-8 70.8 


Table 4: Comparison between ADMM and SOCP/SDP relaxation for the second class of 
test problems with m = 5 and density=0.1. 


Dimension | Algorithm Time (s) 
n=100 ADMM 2.5 
SOCP/SDP | 124 
n=300 ADMM 3.9 
SOCP/SDP 345.6 
n=500 ADMM 5.2 
SOCP/SDP | 1235.8 


Table 5: Comparison between ADMM and SOCP/SDP relaxation for the second class of 
test problems with m = 5 and density=0.01. 


Dimension | Algorithm Time (s) 
n=100 ADMM 2.1 
SOCP/SDP 7A 
n=300 ADMM 3.3 
SOCP/SDP 287.3 
n=500 ADMM 4.5 
SOCP/SDP 933.4 
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Table 6: Results of ADMM for the second class of test problems with m = 5 and 
density=0.001. 


Dimension | KKT1 Time (s) 


n=3000 9.83 x 10-14 37.6 
n=5000 790 <10-% 56.9 
n=8000 See oe ie 110.7 


4 Conclusions 


In this paper, we have applied ADMM for solving the extended trust region 
subproblem. The convergence of the method to the first order stationary con- 
ditions is established and the quality of the solutions for small dimensions are 
compared with the known SOCP/SDP relaxation showing that ADMM con- 
verges to the global solution in significantly shorter time. Moreover, several 
large instances also are solved by ADMM to show its capability. The second 
order convergence analysis of the method is an interesting future research 
direction which one may follow. 
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